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LAYOUT PROCESS FOR MULTIMEDIA MESSAGES 
FIELD OF THE INVENTION 

The present invention is in the field of imaging. The present 
invention relates more particularly to an automatic layout process for a composite 
5 multimedia message, intended to be displayed on a terminal screen, and intended 
to be printed using a printer capable of communicating with the terminal. The 
displayed composite multimedia message can comprise entities of image, text and 
sound (audio). These entities are usually formed from the contents of several initial 
multimedia messages. The automatic layout is performed from an analysis of the 
various initial multimedia messages. Such an analysis especially includes 
characteristics of time, contents, or even context of the images and texts forming 
the initial messages. 

BACKGROUND OF THE INVENTION 

Methods and systems implementing communication means enabling 
multimedia messages to be transmitted and received by adapting their presentation 
have been disclosed in the prior art. International Patent Application WO-01,97504 
describes a method and system of the MMS type (Multimedia Messaging System) 
comprising one or more terminals capable of communicating one with another, and 
linked to one or more printers, via a control means, for example an MMS server. 
Communication means enable the exchange of a multimedia message containing 
for example a digital image and text between a first terminal and a second 
terminal. A request to print the multimedia message, made from the second 
terminal, is controlled (authorized or not, maximum number of copies, etc.) by the 
first terminal transmitting the multimedia message. 

European Patent Application EP-1,085,464 describes means, and 
particularly a method to include additional elements of text or additional figurative 
elements in a digital image capable of being displayed on a te rmin al, in order to 
produce a resulting image including these elements. The disclosed means enable 
the entire multimedia message comprising the resulting image, enriched with the 
additional elements, to be easy and fun to read and to be interpreted once displayed 
on a screen. For example, the covering of certain zones or parts of the image with 
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text, should not remove certain entities of the image contents, and thus 
compromise the overall interpretation or understanding of the displayed message. 
The method described in European Patent Application EP- 1,085,464 enables 
analysis of the digital image, to recognize and identify an "optimal" zone in said 
5 digital image. This optimal zone, also called candidate region, represents the part 
of the image where an additional element (text, figurative element) is placed. The 
candidate regions of the image intended to receive these additional elements are 
regions where the variations of color or gray levels are low, either regions not 
containing entities or main subjects of interest forming the contents of said image. 

10 The method also enables the additional element's placing or color to be modified, 
so as to optimize the overall presentation of the resulting image for an observer. 

European Patent Application EP-1,1 17,230 describes a method and 
a system to present, on a terminal display screen, information coming from 
multimedia components, such as still or animated (video) images, text, or sound 

15 (audio). A presentation model (SMIL: synchronized multimedia integration 
language) is programmed to display a multimedia message according to a 
configuration preset by compilation files, and corresponding to a programmed 
format, so as to synchronize the presentation of the various components forming a 
page of the multimedia message between a terminal transmitting the message and a 

20 terminal receiving said message. This programmed format defines zones where an 
image and a text are displayed respectively, the group forming the multimedia 
message. If the receiving terminal does not have characteristics suited to present 
certain components of the message, these are blanked. 

The means of the prior art do not enable a display of several 

25 multimedia messages to be obtained, for example on a single page. 

SUMMARY OF THE INVENTION 
It is an object of the present invention to provide a process for the 
automatic layout on at least one page and the automatic display on a terminal 
screen of a composite multimedia message formed from several initial multimedia 

30 messages. 
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The process according to the present invention comprises the 
following steps: 

a) manually display, from the terminal keyboard, a group of at least one 
initial multimedia message on the terminal screen; 
5 b) manually select, from the terminal keyboard, at least one initial 

multimedia message from the group of at least one displayed initial multimedia 
message; 

c) manually validate the selection made in step b); 

d) automatically analyze and record sequential data of the at least one 
10 selected initial multimedia message; 

e) automatically analyze and record semantic data of the at least one 
selected initial multimedia message; 

f) automatically analyze and record relational data between the at least one 
selected initial multimedia message; 

15 g) automatically determine, according to recorded data (sequential, 

semantic and relational), at least one transformed multimedia message; 

h) automatically layout on at least one page having a first format, a 
composite multimedia message formed from the at least one transformed 
multimedia message; 

20 i) automatically display the composite multimedia message on the terminal 

screen while keeping the dimensional ratio of the first format; 

. j) manually validate, from the terminal keyboard, the display of the 
composite multimedia message. 

The layout is performed on one or more pages, the number of said 

25 pages being less than or equal to the number of initial multimedia messages. This 
layout is performed especially from an automatic analysis of characteristics 
specific to the initial multimedia messages, each comprising an image, text or 
sound entity. This analysis then enables an optimized layout to be determined on 
one page, or several pages, of a composite multimedia message. The composite 

30 multimedia message is determined according to the key characteristics of the 

contents of each of the initial multimedia messages. This enables not only a display 
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with a single view or a reduced number of views to be obtained, but also the 
printing cost to be reduced, if printing the resulting composite message is selected. 
A single page or even just a few pages are produced and contain all the important 
image and text characteristics of each initial multimedia message. 
5 The composite multimedia message is formed from a selection, 

made by a user from the terminal keyboard, of at least one initial multimedia 
message, and then from an automatic analysis of the contents of the initial 
multimedia message. The number of pages forming the composite multimedia 
message is less than or equal to the number of initial multimedia messages. 
10 The present invention also enables, after the automatic display of 

the composite multimedia message, the display of the proposed composite 
multimedia message to be invalidated manually from the terminal keyboard. In this 
case the invention method re-enables the automatic layout of the composite 
multimedia message on at least one page having a second format, and then enables 
15 the automatic display of this composite multimedia message on the terminal 
screen. And so on, iteratively, so long as the composite multimedia message 
displayed in a given format by the invention method does not suit the user. 

The process of the present invention also enables a printing request 
of the composite multimedia message, validated to a printer that communicates 
with the terminal, to be transmitted from the terminal keyboard. 

Other characteristics and advantages will appear on reading the 
following description, with reference to the drawings of the various figures. 

BRIEF DESCRIPTION OF THE DRAWINGS 
Figure 1 shows an example of a hardware environment used to 
implement the present invention. 

Figure 2 represents a diagram of the operations executed by the 
invention process. 

Figure 3 represents an example of the display of initial multimedia 
messages on a terminal screen. 

Figure 4 represents an example of the display, according to the 
invention process, of a composite multimedia message on a terminal screen. 
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DETAILED DESCRIPTION OF THE INVENTION 

The following description is a detailed description of the main 
embodiments of the invention, with reference to the drawing of each of the various 
figures. 

5 The present invention relates to a process enabling, from one 

terminal 12, the automatic layout, on a single page, or even several pages, of one, 
or more usually, several multimedia messages. Advantageously the number of 
pages is between one and five. The multimedia message is for example, 
advantageously, an MMS message (Multimedia Message Service) comprising a 

10 digital image, the text associated with said image and a sound or audio message 
(music or melody). These multimedia messages can thus be exchanged among 
terminal users functioning in a network. Each of said users being able for example 
to modify the passing message by adding text to it for example. 

Figure 1 represents diagrammatically a preferred embodiment for 

15 implementing the invention. A terminal 12 communicates by a link 17, 1 8 with a 
server 14. The server 14 communicates, by a link 20, with an image database 15. 
The image database 15 stores or saves for example previously recorded high- 
resolution digital images. The server 14 communicates, by a link 19, with a printer 
16. The printer 16 is for example a printer using specific paper (support) enabling 

20 it to produce high quality images; this type of printer, for example inkjet, thermal, 
laser or photographic, is commonly used in digital photographic processing 
laboratories (minilabs). Other image supports, used in online printing services, can 
be used: T-shirts, identity photographs, etc. The terminal 12 is preferably a mobile 
terminal: for example a cellphone equipped with a keyboard 19A and a display 

25 screen 1 8 A, a cellphone equipped with a camera module (phone cam) enabling 

digital shots to be recorded, or a digital camera equipped with a GSM/GPRS or 3G 
type cell modem (wireless cam) enabling digital photographs to be downloaded 
from a remote online album, hi the preferred embodiment where the terminal 12 is 
a mobile terminal which operates for example on a GSM type network (Global 

30 System for Mobile), a gateway 13 enables communications between this terminal 
12, the server 14 and the image database 15 to be administered. The gateway 13, 



WO 2004/019603 PCT/EP2003/008241 

6 

for example of WAP type (Wireless Application Protocol) enables communication, 
represented by the link 17, between the terminal 12 and the network, for example 
GSM. The present invention can be implemented in an Internet environment. That 
is, in a preferred embodiment using a mobile terminal 12 for example of "phone 
5 cam" type, this terminal 12 is capable of communicating with an Internet type 
network. A user who wishes for example to access from the terminal 12, one or 
more images contained in the database 15 linked for example, via the link 1 8, to 
Internet, transmits a message in the appropriate protocol, for example WAP, 
intended for a telephone line. The message transits by the gateway 13, where it is 
10 transformed into a message according to the HTTP protocol (Hypertext Transfer 
Protocol), used in the Internet. Thus the user can recover and display on their 
terminal screen, one or more images coming from the database 15. 

In a particular embodiment, if the terminal 12 is for example of the 
"phone cam" type, the shots are taken directly from the terminal 12, and saved or 
15 stored in a memory of said terminal 12. In this case, the images have for example a 
VGA type resolution (Video Graphics Array) of 640 by 480 pixels. 

According to Figure 2, a user of the terminal 12 manually displays, 
from the terminal keyboard 12, one, or usually several, initial multimedia 
messages. An initial multimedia message comprises a digital image, and either at 
20 least one text message and/or one sound message. This sound message comprises 
audio data, for example known music or a melody. The digital image contained in 
the multimedia message can be still or animated (video clip). The audio data are 
saved in sound files, for example on the server 14. According to the size and 
capacity of the display screen of the terminal 12, one or more messages are 
25 displayed on one page of said screen. 

In a first embodiment, the method of displaying the initial 
multimedia messages can be reduced to the display of successive lines, giving 
simply: "message 1", "message 2", etc. By clicking on "message 1", the 
corresponding multimedia message then appears on the screen in the form for 
30 example of an image (image zone 25, 26, 27, 28) under which one or more texts 
appear (text zone 29, 30, 31, 32). The text zone 29, 30, 31, 32 can also appear 
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above or beside the image zone. The text zone 29, 30, 3 1, 32 explicitly gives one 
or more texts inherent to the image; this is case when the texts are relatively short, 
that is are only composed of a few words or expressions. Or the text zone 29, 30, 
31, 32 gives lines of the type "text 1", "text 2", etc., and the user explicitly makes 
Hie corresponding text appear by clicking "text 1", "text 2", etc. 

In a second embodiment, and according to Figure 3, initial 
multimedia messages 21, 22, 23, 24 appear, one under the other, as a page 20A 
displayed on the screen 1 8 A. The page of screen 20 has for example four 
messages. Each of these initial multimedia messages 21, 22, 23, 24 comprises an 
image zone 25, 26, 27, 28, a text zone 29, 30, 31, 32, and a sound zone 32A, 33; 
the sound zone 32A, 33 indicates that a sound message is associated with said 
initial multimedia message 22, 24. Each of the images 25, 26, 27, 28 is displayed 
for example with low resolution enabling the contents of said image to be viewed. 
The text zone can show, as in the first embodiment presented above, either a 
shortened text (e.g. "text 1", "text 2", etc.), or the explicit display of the text 
associated with the image of the initial multimedia message. In the special case 
where a sound message is associated with the initial multimedia message 22, 24, 
an icon or distinguishing symbol 32A, 33 is displayed; this distinguishing symbol 
is for example a musical note. If the user wishes to select more than four messages, 
they use the keyboard 1 9A to scroll the other initial multimedia messages. 

According to Figures 2 and 3, the user manually selects, using the 
keyboard 19A of terminal 12, one or more of these displayed messages from the 
display of these initial multimedia messages 21, 22, 23, 24. According to a 
particular embodiment, the selection is made using the keyboard 19A of the 
terminal 12, for example by pointing to the selected messages 21, 23, 24, by 
clicking on a special zone 34, 35, 36 specific to each of said messages. The 
selected messages 21, 23, 24 make up for example a story, said messages having 
been transmitted successively in time by different transmitters. When the user has 
selected a group of at least one initial multimedia message, they validate the 
selection 1 thus made, by using for example a menu specific to the invention 
method, and which is displayed on the screen 18A of the terminal 12. An 
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instruction, for example of the type "message for layout", is proposed in this menu. 
The invention process then enables the automatic analysis of the initial multimedia 
messages selected for layout. The resulting data of this analysis are automatically 
saved. The analysis is performed by the algorithm used to implement the invention 
process. This algorithm comprises a series of instructions represented 
diagrammatically in Figure 2. In a preferred embodiment, the application or 
program specific to the invention process is installed on the terminal 12; in this 
embodiment for example a terminal of the type NOKIA™ 7650 with code 
developed in Java or C++ is used. But, in another embodiment, the program can 
also be installed on the server 14. A first step of the analysis consists in 
automatically performing a sequential analysis 2 of each selected and validated 
initial multimedia message. This sequential analysis 2 enables the characterization, 
in time of the data corresponding to the moment when the initial multimedia 
message is created, the moment when the message is sent, the moment when the 
message is received, the identification of the transmitter of the message, the 
identification of the receiver of said message. According to this embodiment, the 
analyzed sequential data are saved either to a memory of the terminal 12, or to a 
memory of the server 14. 

A second step of the analysis consists in automatically performing a 
semantic analysis 4 of the initial multimedia message. This semantic analysis is an 
analysis of the entire contents of the initial multimedia message: Image, text, sound 
or voice inherent to the message contents. The algorithm of the invention method 
enables for example only part of an initially rather long text to be kept for later 
layout, while keeping the meaning of the text, using keywords of said text. 
According to this embodiment, the analyzed semantic data are saved either to a 
memory of the terminal 12, or to a memory of the server 14. 

In another embodiment of the invention, the step of automatic 
analysis and recording of semantic data is performed before the step of automatic 
analysis and recording of sequential data. 

A third step of the analysis consists in automatically performing a 
relational analysis 3 between the selected initial multimedia messages. The 
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algorithm of the invention process enables links or correspondences to be 
established between images, texts and sounds forming each of the various selected 
initial multimedia messages 21, 23, 24 respectively. The relational analysis is 
performed using all the previously saved sequential and semantic data. This 
5 relational analysis enables relational data to be established that correspond to links 
or correspondences that exist between the respective contents of each of the 
selected initial multimedia messages, and are linked to the respective meaning of 
each of said messages; this enables the automatic determination, by using the 
sequential and semantic data, of the relational data that weigh the existing 

10 relationship level between each of the selected initial messages. The relational data 
are saved either to a memory of the terminal 12, or to a memory of the server 14. 

The algorithm of the invention process uses all the saved sequential, 
semantic and relational data to automatically determine at least one transformed 
multimedia message 38, 39, 40, 41, 42, 43. According for example to the existing 

15 relationship level between two initial multimedia messages, the algorithm of the 
invention process enables resizing, zooming, only keeping one region or zone of 
interest (e.g. a face) of the respective images of each of said initial messages, 
and/or only keeping one part of the text or text parts forming said initial messages. 
La an embodiment where the multimedia message comprises an image that is a 

20 video clip, the invention process enables the determination for example of a key 
image of said video clip; this key image is then processed by the invention process 
to be included in a composite multimedia message. 

The invention process thus enables the determination of a 
multimedia message that is a transformation of the initial multimedia message, 

25 because only the key parts of the initial messages are kept. This transformation 6 

has a predictive objective, aiming especially to optimize later layout. Naturally, not 
all the selected initial multimedia messages 21, 23, 24 are necessarily transformed. 
The transformed multimedia message 38, 39, 40, 41, 42, 43 can be identical, in 
overall contents (image, text and sound), to the selected initial multimedia message 

30 21, 23, 24. After this transformation step, the number of transformed multimedia 
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messages is equal to the number of initial multimedia messages. All said 
transformed messages constitute the composite multimedia message. 

According to Figure 4, the invention process then automatically lays 
out 7, on at least one page 37 having a given format, the composite multimedia 
5 message, formed from each transformed multimedia message. Figure 4 illustrates 
for example a layout of six selected multimedia messages 38, 39, 40, 41, 42, 43. 
Then the composite multimedia message is displayed automatically 8 on the 
terminal screen 12, according to a format given by the invention process. Figure 4 
illustrates an example in which the invention process lays out six multimedia 

10 messages on a single page 37. This means that either only six multimedia messages 
were previously selected and validated, and then laid out on this page 37, or a 
larger number of messages, for example ten, twelve or eighteen were selected and 
validated. If ten messages are selected and validated, the invention process, 
according to the automatic analysis of these messages, and programmed formats, 

15 lays out and displays for example six messages on a first page 37, and then four 
messages on a second page 37. If twelve messages are selected and validated, the 
invention process, according to the automatic analysis of these messages, and 
programmed formats, lays out and displays for example six messages on a first 
page 37, and then six messages on a second page 37. If eighteen messages are 

20 selected and validated, the invention process, according to the automatic analysis 
of these messages, and programmed formats, lays out and displays for example six 
messages on a first page 37, six messages on a second page 37, and six messages 
on a third page 37. When the composite multimedia message is laid out and 
displayed on several pages 37, the display of each of said pages 37 on the screen 

25 18, can be performed manually, by using the keyboard 1 9 A of the terminal 12. But, 
in an advantageous embodiment of the invention, the display of several pages 37 is 
performed automatically according to a set display time, for example ten or fifteen 
seconds display per page. The display time is advantageously set for each page, or 
the display time is set globally for a group of pages, each of said pages being 

30 displayed automatically for a time virtually equal to a sub-multiple of the global set 
time. The display time can be modified from the keyboard 19 A. The format, 
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characterized by its width 44 and its height 45, is selected automatically by the 
invention method, according to initially programmed format data. The size ratio 
characterizes for example the ratio between the height and width of the format. The 
composite multimedia message is displayed as an image (image format), that is as 
5 a bitmap file (pixels). According to the drawings illustrating Figure 4, and 

according to messages 38, 39, 40, the displayed image represents for example one 
or more people's faces, and the texts that correspond to verbal exchanges among 
said people are displayed in specific zones 46, 47, 48, 49; these zones 46, 47, 48, 
49 are represented for example, in the image, as balloons. If the displayed 

10 composite multimedia message 37 suits the user of the terminal 12 having 

requested the layout, said user manually validates 10 the proposed display, from 
the keyboard 19 of the terminal 12. The invention process also enables the user, 
from for example a menu displayed on the screen of the terminal 12, to perform a 
printing request 1 1 of the validated composite multimedia message. According to 

15 Figure 1, a printer 16 produces the composite multimedia message, for example on 
a sheet of photographic paper, in a format whose size ratio is identical to the size 
of the automatic layout. The invention process thus enables a number of printing 
sheets less than or equal to the number of initial multimedia messages to be 
produced, all said messages telling for example a story. Savings of printing paper 

20 are thus advantageously made. 

Li a particular embodiment, the invention process also enables the 
transformed multimedia message to determined by using analysis rules 5 that 
depend on a context linked to the multimedia message. These analysis rules 5 are 
included in the algorithm, and enable instructions to be executed that add to the 

25 semantic analysis of the message, and that influence the presentation of the 

transformed multimedia message, and thus the layout, according to the nature of 
said message: professional context (property, insurance, police, etc.), holidays, 
family, night-environment, etc. The analysis rules are worked out according to the 
most likely events occurring in each context. These most likely events enable for 

30 example the characterization of the image entities to be kept in order to determine 
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the transformed multimedia message, or for example the keywords to be kept in 
order to form the texts of the transformed multimedia message. 

The object of the algorithm of the invention process is to optimize 
the layout of the group of selected initial multimedia messages, in order to form a 
5 composite multimedia message, according to the available programmed formats. 
However, if the user decides to invalidate the layout, the invention process enables 
another layout 9 to be performed automatically, by using a second format, different 
than the first format corresponding to the invalidated layout. The user has the 
choice of validating 10 this second layout, or continuing to invalidate it. The 

10 invention process enables a third layout to be performed automatically, according 
to a third format different than the first and second formats. Invalidation can be so 
repeated, iteratively, so long as different programmed formats are available that 
have not been used to perform an automatic layout. In other words, if the number 
of programmed formats in the invention process is "n", "n" being an integer, the 

15 invention process enables the layouts automatically determined by the invention 
process to be invalidated "n" times. Once all the available programmed formats 
have been invalidated, the invention process proposes the first invalidated format 
to the user again, then the second, etc. In this case, the user only has the option of 
validating the format, from the "n", that suits them best. In the preferred 

20 embodiment of the invention, the number of programmed formats is between one 
and ten. However, there is no upper limit of "n" to implement the invention 
process. 

While the invention has been described with reference to its 
preferred embodiments, it is clear that variants and modifications can be produced 
25 within the scope of the claims. 



